Implement bot detection and verification by jazairi · Pull Request #362 · MITLibraries/timdex-ui

jazairi · 2026-02-26T17:08:08Z

Why these changes are being introduced:

We need a preliminary solution to manage bot traffic in production.

Relevant ticket(s):

https://mitlibraries.atlassian.net/browse/USE-410

How this addresses that need:

Implements a Bot Detector service that uses Crawler Detect to identify bots to refer to Cloudflare Turnstile.
Implements a Turnstile Controller using the rails-cloudfire-turnstile gem for verification challenges.

Side effects of this change:

The new form may need additional styling. For now, I just added our button styles.

Developer

Accessibility

ANDI or WAVE has been run in accordance to our guide.
This PR contains no changes to the view layer.
New issues flagged by ANDI or WAVE have been resolved.
New issues flagged by ANDI or WAVE have been ticketed (link in the Pull Request details above).
No new accessibility issues have been flagged.

New ENV

All new ENV is documented in README.
All new ENV has been added to Heroku Pipeline, Staging and Prod. (Added to GeoData and TIMDEX UI prod only)
ENV has not changed.

Approval beyond code review

UXWS/stakeholder approval has been confirmed.
UXWS/stakeholder review will be completed retroactively.
UXWS/stakeholder review is not needed.

Additional context needed to review

This introduces rails-cloudflare-turnstile and crawler_detect as dependencies.

Code Reviewer

Code

I have confirmed that the code works as intended.
Any CodeClimate issues have been fixed or confirmed as
added technical debt.

Documentation

The commit message is clear and follows our guidelines
(not just this pull request message).
The documentation has been updated or is unnecessary.
New dependencies are appropriate or there were no changes.

Testing

There are appropriate tests covering any new functionality.
No additional test coverage is required.

jazairi · 2026-02-26T17:11:28Z

Looks like the coveralls outage is ongoing. I'd like to wait until that's resolved to merge, but I think we can start code review in the meantime.

JPrevost

Can you configure the PR build to have this feature enabled and functioning? I want to easily be able to confirm my normal user agent is not blocked from anything but if I change my user agent to a known bot I should be challenged and then be able to pass the challenge to continue.

config/initializers/cloudflare_turnstile.rb

app/controllers/turnstile_controller.rb

JPrevost · 2026-03-02T14:22:20Z

app/controllers/turnstile_controller.rb

+  private
+
+  def ensure_bot_detection_enabled
+    head :not_found unless Feature.enabled?(:bot_detection)


If we are assuming this should never happen, maybe we should log to Sentry if it does? My assumption is this is handling a case in which a user agent is requesting the turnstile route even though we have it disabled which should not happen under normal circumstances.

That's the idea, but now I'm wondering if we even care whether user agents are requesting the turnstile route. I might just drop this for clarity's sake.

app/models/bot_detector.rb

qltysh · 2026-03-02T18:03:18Z

❌ 2 blocking issues (2 total)

Tool	Category	Rule	Count
rubocop	Lint	Class has too many lines. [289/100]	1	❌
rubocop	Style	Line is too long. [121/120]	1	❌

Gemfile

app/models/bot_detector.rb

qltysh · 2026-03-02T18:03:20Z

app/models/feature.rb

 class Feature
  # List of all valid features in the application
-  VALID_FEATURES = %i[geodata boolean_picker oa_always simulate_search_latency tab_primo_all tab_timdex_all
+  VALID_FEATURES = %i[bot_detection geodata boolean_picker oa_always simulate_search_latency tab_primo_all tab_timdex_all


Line is too long. [121/120] [rubocop:Layout/LineLength]

test/models/bot_detector_test.rb

Why these changes are being introduced: We need a preliminary solution to manage bot traffic in production. Relevant ticket(s): - https://mitlibraries.atlassian.net/browse/USE-410 How this addresses that need: - Implements a Bot Detector service that uses Crawler Detect to identify bots to refer to Cloudflare Turnstile. - Implements a Turnstile Controller using the `rails-cloudfire-turnstile` gem for verification challenges. Side effects of this change: The new form may need additional styling. For now, I just added our button styles.

coveralls · 2026-03-02T18:41:06Z

Pull Request Test Coverage Report for Build 22869511702

Details

37 of 37 (100.0%) changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.05%) to 98.188%

Totals
Change from base Build 22866584976:	0.05%
Covered Lines:	1355
Relevant Lines:	1380

💛 - Coveralls

- Explicity require Feature model in initializer - Update CrawlerDetect method name - Remove trailing whitespace from Bot Detector test - Switch from OpenStruct to Struct - Update test assertions Test Turnstile root path fallback and fix typo in CrawlerDetect instantiation

jazairi · 2026-03-02T20:51:09Z

@JPrevost I think this is ready for review again. I pushed two separate commits, one two respond to code review feedback, and the other for linting and cleanup. When I enabled the feature in the PR build, I learned that my method name for CrawlerDetect was wrong, so that's fixed. 🤦 I'd flipped the bots? boolean to test locally, and all the stubs had the wrong name, too -- a good case for VCR, perhaps.

In any case, here's how I've tested the feature:

Set my UA to Googlebot in dev tools.
Run a search and confirm that the challenge view shows.
Update your UA to something non-bot and confirm that you can pass the challenge. (In my experience, it does not let you pass the challenge if using a bot UA.)

JPrevost

I think there may still be an file remnant from the initial implementation?

Some of the other comments can either be discussed, adjusted, or opened as follow-on tickets depending on how you'd prefer to address them.

app/controllers/turnstile_controller.rb

app/views/turnstile/new.html.erb

config/routes.rb

test/controllers/search_controller_test.rb

config/initializers/cloudflare_turnstile.rb

jazairi · 2026-03-09T18:17:33Z

@JPrevost Thanks for the detailed review on this! Let me know if I missed anything on the latest commit. I took an initial pass at reviewing the language, but I'm not sure if UXWS should get eyes on it as well?

test/controllers/turnstile_controller_test.rb

JPrevost

Looks good.

Please open a ticket with screenshots for UX to review the language once we land this. It can be a backlog ticket as the wording seems good enough for now.

JPrevost

Sorry for overriding my last review.

Please check if the CI failure is real before we merge.

- Clean up unused view - Avoid mocks for bot detection - Additional guardrails for initializer - Soften language

JPrevost

See previous note about a ticket for UX with screenshots to review the language. Thanks for your patience on the review!

jazairi force-pushed the use-410-turnstile-gem branch from d4f048e to 460d738 Compare February 26, 2026 17:09

jazairi had a problem deploying to timdex-ui-pi-use-410-tu-bsn4kh February 26, 2026 17:09 Failure

jazairi requested review from JPrevost and matt-bernhardt February 26, 2026 17:10

JPrevost self-assigned this Feb 27, 2026

JPrevost requested changes Mar 2, 2026

View reviewed changes

qltysh bot reviewed Mar 2, 2026

View reviewed changes

jazairi had a problem deploying to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 18:09 Failure

jazairi added 2 commits March 2, 2026 10:11

Respond to code review feedback

3c1b8e0

jazairi force-pushed the use-410-turnstile-gem branch from 76f6402 to fb87d65 Compare March 2, 2026 18:39

jazairi temporarily deployed to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 18:41 Inactive

jazairi force-pushed the use-410-turnstile-gem branch from fb87d65 to 97274a2 Compare March 2, 2026 18:41

jazairi temporarily deployed to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 18:43 Inactive

jazairi temporarily deployed to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 18:46 Inactive

jazairi force-pushed the use-410-turnstile-gem branch from 42b13a4 to 5c844c9 Compare March 2, 2026 19:32

jazairi temporarily deployed to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 19:33 Inactive

jazairi force-pushed the use-410-turnstile-gem branch from 5c844c9 to f04ae32 Compare March 2, 2026 19:36

jazairi temporarily deployed to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 19:36 Inactive

jazairi force-pushed the use-410-turnstile-gem branch from f04ae32 to bf6be71 Compare March 2, 2026 19:37

jazairi temporarily deployed to timdex-ui-pi-use-410-tu-bsn4kh March 2, 2026 19:37 Inactive

JPrevost requested changes Mar 9, 2026

View reviewed changes

jazairi force-pushed the use-410-turnstile-gem branch from 1cb7ab3 to 5e46548 Compare March 9, 2026 18:16

jazairi requested a review from JPrevost March 9, 2026 18:16

jazairi force-pushed the use-410-turnstile-gem branch from 5e46548 to 94a2123 Compare March 9, 2026 18:19

qltysh bot reviewed Mar 9, 2026

View reviewed changes

test/controllers/turnstile_controller_test.rb Outdated Show resolved Hide resolved

test/controllers/turnstile_controller_test.rb Show resolved Hide resolved

test/controllers/turnstile_controller_test.rb Show resolved Hide resolved

JPrevost approved these changes Mar 9, 2026

View reviewed changes

JPrevost requested changes Mar 9, 2026

View reviewed changes

Address code review feedback

a01ead6

- Clean up unused view - Avoid mocks for bot detection - Additional guardrails for initializer - Soften language

jazairi force-pushed the use-410-turnstile-gem branch from 94a2123 to a01ead6 Compare March 9, 2026 18:50

JPrevost approved these changes Mar 9, 2026

View reviewed changes

jazairi merged commit 0ef2f65 into main Mar 9, 2026
6 checks passed

jazairi deleted the use-410-turnstile-gem branch March 9, 2026 19:05

Conversation

jazairi commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why these changes are being introduced:

Relevant ticket(s):

How this addresses that need:

Side effects of this change:

Developer

Accessibility

New ENV

Approval beyond code review

Additional context needed to review

Code Reviewer

Code

Documentation

Testing

Uh oh!

jazairi commented Feb 26, 2026

Uh oh!

JPrevost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JPrevost Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

jazairi Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

qltysh bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 blocking issues (2 total)

Uh oh!

Uh oh!

Uh oh!

qltysh bot Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 22869511702

Details

💛 - Coveralls

Uh oh!

jazairi commented Mar 2, 2026

Uh oh!

JPrevost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jazairi commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JPrevost left a comment

Choose a reason for hiding this comment

Uh oh!

JPrevost left a comment

Choose a reason for hiding this comment

Uh oh!

jazairi commented Feb 26, 2026 •

edited

Loading

qltysh bot commented Mar 2, 2026 •

edited

Loading

coveralls commented Mar 2, 2026 •

edited

Loading

jazairi commented Mar 9, 2026 •

edited

Loading